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Abstract 

We present a fast implementation of an electronic 
digital image stabilization system that is able to handle 
large image displacements. The system has been im- 
plemented in a parallel pipeline image processing hard- 
ware (Datacube Max Video 200) connected to a SUN 
SPARCstation 20/612. Our technique is based on a 2D 
feature-based multi-resolution motion estimation algo- 
rithm, that tracks a small set of features to estimate 
the motion of the camera. The combination of the 
estimates from a reference frame is used to warp the 
current frame in order to achieve stabilization. Ex- 
perimental results using video sequences taken from a 
camera mounted on a moving vehicle demonstrate the 
robustness of the system when processing 15 frames per 
second. 



on a mosaic-based registration technique using a pyra- 
midal hardware (VFE-100). The system uses a multi- 
resolution, iterative process to estimate the affine mo- 
tion parameters between levels of Laplacian pyramid 
images. From coarse-to-fine levels, the optical flow of 
local patches of the image is computed using a cross- 
correlation scheme. The motion parameters are then 
computed by fitting an affine motion model to the flow. 
These parameters are used to warp the previous image 
of the next finer pyramid level to the current image, 
and the refinement process continues until the desired 
precision is achieved. This scheme, combined with the 
construction of a mosaic image, allows the system to 
cope with large image displacements. The VFE im- 
plementation is capable of stabilizing images of size 
128 x 120 pixels, with image displacements ranging 
from ±32 to ±64 pixels, at a rate of 10 frames per 
second [5]. 



1. Introduction 

Electronic digital image stabilization is the process 
of removing the unwanted motion from an input video 
sequence by appropriately warping the images. Im- 
age stabilization has been used for the computation 
of egomotion [11, 6], video compression [7], detection 
of IMOs [2, 9], and tracking of IMOs [1]. For more 
natural visualization, vehicle models allow the filtering 
of the high frequency or oscillatory motion due to ir- 
regularities of the terrain from the desired "smooth" 
motion that would result if the vehicle were running- 
over smooth terrain [4, 12]. 

Fast implementations of electronic image stabiliza- 
tion algorithms are presented in Hansen et al. [5], and 
Morimoto ti al [9]. Hansen et al. [5] describe the 
implementation of an image stabilization system based 
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We present a fast and robust implementation of a 
digital image stabilization algorithm that is based on 
the 2D model described in [13]. A prototype of this 
system was presented in [9]. Both systems were im- 
plemented in a Datacube Max Video 200, a machine 
commonly used for real-time image processing. The 
prototype is able to stabilize 128 x 120 pixels at a rate 
of 7 frames per second, with image displacements of 
up to ±15 pixels. Improvements on the algorithm al- 
lowed us to double the frame rate to approximately 15 
frames per second using the same resolution images, 
with image displacements of up to ±21 pixels between 
consecutive frames. This performance, as discussed in 
Section 3, is similar to that presented in [5]. 

This paper is organized as follows. Section 2 intro- 
duces the image stabilization algorithm and describes 
how it was implemented. Section 3 shows experimental 
results of the performance of the system, comparing it 
with other existing systems, and Section 4 concludes 
this paper. 



2. Electronic Image Stabilization 

Our electronic digital image stabilization algorithm 
is composed of two modules. The motion estimation 
module computes the motion between two consecutive 
frames by tracking a small set of feature points and 
then fitting a 2D model to the displacements of the 
features. After the interframe motion is estimated, the 
motion parameters are sent to the second module, the 
motion compensation module, which keeps a history of 
the whole movement of the camera in order to "stabi- 
lize" the current frame. 

2.1. Motion Estimation 

We use a 2D rigid motion model [13] that allows 
translation, rotation around the optical axis and scal- 
ing of the image frames. The transformation between 
two image frames is defined by: 
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where (A,-, Yi) are the image frame coordinates at time 
U for i = {0, 1}, (A-Y AY)' is the translation vector 
measured in the image coordinate system of the frame 
at t Q (/o), 6 is the rotation angle between the two 
frames and S is the scaling factor. Notice that S is 
inversely proportional to the ratio of the distances be- 
tween two arbitrary image points at times t 0i t x . Thus 
S can be computed given a set of matched points from 
both frames, independently of the translation and ro- 
tation between them. 

To estimate the motion parameters given a set 5, 
of N feature points in frame /; and the set Sj of cor- 
responding points in frame /,-, the scaling factor S is 
estimated first by computing the ratio of the distances 
in the feature sets relative to their center of mass [13]. 
Assuming small rotation, the trigonometric terms in 
(1) can be linearized to compute the remaining transla- 
tion and rotation parameters. A system of linear equa- 
tions is then obtained by substituting all N matched 
feature pairs into the linearized equations. Each pair 
introduces two equations, hence the linear system has 
2N equations and three unknowns (0, AX, and AY), 
which is solved by a least squares method. 

2.2. Automatic Feature Tracking 

Before computing the motion parameters, we must 
acquire and track a small set of feature points. The 
problem with this approach is that the selected fea- 
tures must belong to regions that suit the 2D motion 



assumption. In general, distant scene points are appro- 
priate. A heuristic rule to select features on the horizon 
is currently used. The horizon is a very strong visual 
cue that is also used by other stabilization techniques 
[3,4, 12]. 

Feature tracking is performed by a multi-resolution 
scheme. The initial step is to construct a Laplacian 
pyramid for both frames. A Laplacian pyramid L[t] is 
formed by combining several reduced resolution Lapla- 
cian images of frame /«. Each level of the pyramid will 
be denoted by L[t t l\ t and L[t t 0) will have the Lapla- 
cian image of the original frame of size Z (an image 
of size Z has Z x Z pixels). The Laplacian image is 
formed by smoothing the input image with a Gaussian 
kernel and then convolving the smoothed image with 
a Laplacian kernel operator. The size of an arbitrary 
pyramid level / will be given by The levels of the 
pyramids L[t - 1] and L[t] are used from coarse to fine 
resolutions, where each new processed level contributes 
to refining the position of the features. 

A match within an arbitrary level is obtained by 
minimizing the sum of square differences (SSD) over 
a neighborhood (search window) around the candidate 
matches in frame / t . The selection of features is per- 
formed by thresholding L[t- 1, 0]. This thresholded im- 
age is divided into N vertical partitions, where A r cor- 
responds to the number of features to be tracked. Each 
partition is searched from top to bottom and the top 
most feature is selected for tracking. The features are 
scaled-down to the coarsest resolution level L[t - I J] 
and used to determine their matches in L[tJ]. For 
a feature /^(r.y), a search for the minimum SSD is 
performed in a window of size 5= (2s+l)x(2s+l) cen- 
tered at the pixel P{(x t y). Let P/(u, v) be the selected 
matching point of P}_ x {x,y), Notice that the maxi- 
mum displacement supported by this search is only s 
pixels. For the next pyramid level, the coordinates of 
these pairs are scaled- up by a factor of 2. For the fea- 
ture P/JT 1 l (2ar,2t/), the search for the minimum SSD is 
now performed around the pixel P/" l (2u, f 2v). This 
process is repeated until the level 0 is reached. Notice 
that since the displacement is doubled after every level, 
the total displacement that this algorithm can handle 
can be very large even for small values of s. 

The use of Laplacian images for feature tracking also 
helps to reduce the sizes of the SSD windows. The 
Laplacian operator enhances the regions around the 
tracked features and smoothes the regions of constant 
brightness that in general surround the edges. Since 
the Laplacian operator is rotation invariant, edges of 
different orientations are equally enhanced so that the 
Laplacian images are not affected by possible rotations 
of the input images. 



' 2.3. Motion Compensation 

i'. 

The motion compensation module keeps a history 
of the interframe motion to remove what is unwanted 
and compute the warping parameters that will stabi- 

1 lize the current image frame. One of the advantages of 

electronic image stabilization systems is that motion 
can be compensated on demand, offering great flexibil- 
ity by simply modifying some parameters of the com- 
pensation module. The way motion is estimated and 

I; compensated defines the structure of the stabilization 

! system. 

ji One possible way to stabilize an image stream is 

to keep a reference frame and compute the motion of 
the current frame relative to that reference. The prob- 
lem with this approach is that the overlapping region 
will eventually become insignificant, hence impossible 
to compute the motion parameters. We will describe 
an algorithm that computes the motion parameters be- 
tween two consecutive frames, which are likely to have 
good overlapping, thus avoiding the previous problem. 

i< Although a reference is still necessary, the system is 

!| able to compute the interframe motion even when the 

1 reference becomes obsolete. 

I The block diagram of the stabilization algorithm is 

| shown in Figure 1. The Laplacian L[t] is built from 

the input image and sent to a delay buffer storage that 
keeps L[t — 1] for the estimation process. The feature 
detection /tracking module uses L[t - 1,0] to extract 
features which are used for tracking. After the grid-to- 
grid matches are found, the motion estimation module 
computes the interframe motion. The motion compen- 
sation block must then compute the total motion of the 
camera by combining all the interframe estimates over 
i time (since it started running, e.g., t = 0). The total 

i motion is used to warp the current frame and the 

result is sent to the monitor. Since all frames of the se- 
quence are being warped back to £[0,0], the resulting 
,{ sequence is stabilized. 

Since features are tracked between consecutive 
frames, they do not need to be extracted in every frame. 
They can be extract every other frame instead. Sup- 
pose that the system uses the frame /, to extract fea- 
tures. This set can be used for tracking features back- 
wards between f t and /t-i. When / t+1 is acquired, 
the same set can be used for forward tracking between 
ft and /t+i. Finally, when /* + 2 comes in, new features 
must be extracted and the process continues with back- 
ward tracking. 

This strategy is very simple to implement and just 
ij requires a slightly more complex control structure. The 

forward tracking is done just like described previoxisly, 
;j but a few changes must be made for backward tracking. 
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Figure 1. Block diagram of the stabilization algo- 
rithm 



First, the same tracking algorithm is used by simply 
switching /, with / t _i and the motion based on the 
inverted pairs is computed. The result of this process 
estimates the motion from f t to ft-i- A simple trans- 
formation is used to obtain the forward motion from 
the backward motion. 

If the motion can be reliably estimated for large fea- 
ture displacements, it is very simple to extend the back- 
ward/forward estimation scheme to utilize larger steps 
between feature extractions. 

Due to the dynamic nature of the problem, the scene 
as a whole is under constant modification, so that af- 
ter some relatively short time the original frame might 
not have anything in common with the current frame. 
For display purposes, the user should then intervene 
and reinitialize the system, so that another reference 
is taken and displayed. This process can be made au- 
tonomous by setting limits over the motion parameters, 
so that whenever the reference frame becomes obsolete, 
the system can reset itself automatically. 

To obtain better visualization for lateral camera mo- 
tion, mosaic images can be constructed. When the 
camera moves predominantly parallel to its optical 
axis, the scaling factor $ may not be compensated for, 
what produces an output sequence that looks derotated 
only [3]. The smoothing techniques suggested by [4, 12] 
could be applied for the remaining parameters. 

The system was implemented using a Max Video 200 
board connected to a SUN SPARCstation 20/612 via 
a VME bus adaptor. The MV200 is a parallel pipeline 
image processing hardware system manufactured by 
Datacube Inc., very commonly used for real-time image 
processing. 



3. Experimental Results 

This section presents experimental results obtained 
from a video sequence recorded from a camera rigidly 
mounted on a moving vehicle. Unfortunately, still im- 
ages are not the most appropriate way of displaying 
the results of a dynamic process such as stabilization. 
Those readers with access to www browsers might have 
a look at http://www.cfar.umd.edu/ "carlos/ sta- 
bilization. html, where a few example sequences are 
available in MPEG format. 
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Figure 2. Stabilization results for off-road naviga- 
tion. 



The upper left image of Figure 2 shows frame F of 
the original sequence. The upper right corner corre- 
sponds to the stabilized image of that frame F t . The 
bottom left image shows the difference between frame 
F t and its previous frame from the original se- 
quence, and the bottom right shows their difference 
after stabilization. Jn both cases, only the pixels with 
absolute difference higher than 10 are displayed. The 
dense black region around the difference of stabilized 
frames are due to the compensation of motion (the im- 
age warping performed to register frame F t to the refer- 
ence frame). Since the algorithm registers the current 
frame to the reference, the difference of images can be 
considered as an error measure that stabilization at- 
tempts to minimize. 

Observing the image differences, it is clear that the 
error is quite large in the regions that are closer to the 



camera, but it is considerably reduced in the regions 
around the horizon, where the features are extracted 
and tracked. Actually, the regions closer to the camera 
do not fit the 2D motion computed from the tracking 
of features on the horizon, but still, the system is able 
to process the sequence. 

We can compare the performance of our system with 
the VFE based system presented in [5] by considering 
the case of pure image translation with constant ve- 
locity. In such a case, the VFE system supports ve- 
locities of 320 pixels per second, while our Datacube 
based system supports 313 pixels per second. These fig- 
ures are important when stabilizing off-road sequences, 
but sometimes not high enough. The robustness of 
both systems to support higher displacements can be 
increased by enlarging the search windows, but with 
loss in frame-rate speed. The table in Figure 3 shows 
how the frame-rate of our datacube implementation de- 
grades when the search window is increased and the 
SSD windows are kept constant (7x7 pixels). Local 
windows of size 3 pixels allows global displacements of 
size 21 (at the finest resolution). At this settings the 
system can process 14.9 frames per second. The last 
row of the table shows the performance of the VFE 
implementation and the right-most column gives the 
maximum pixel velocities that each setting is able to 
process. This might suggest that our system is more 
robust when set to search displacements of magnitude 5 
instead of 3 because despite the loss in frame-rate there 
is a considerable gain in feature velocity that it is able 
to track. Unfortunately, when the search space is in- 
creased, the narrow SSD window being used is more 
likely to find a false match. Increasing* the SSD win- 
dow provides better discrimination between features 
and decreases the probability of false matches. On the 
other hand the frame- rate drops considerably. 
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Figure 3. Performance evaluation of search win- 
dow sizes. 



One of the advantages of our approach is that the 
matches are restricted to distant points and not blocks 
distributed all over the image like the VFE system. 
This allows our system to work under situations where 
regions, which are not used to track features, present 
very intense flow that do not fit the 2D rigid mo- 
tion assumption. When the flow is due to dominant 



translation, the displacement of distant features will be 
smaller than the displacement of close features. Thus 
using a subset of feature pairs with the smallest dis- 
placements to estimate the motion parameters con- 
tributes to enhance the robustness of the system. A 
possible alternative to the current feature extraction 
and selection algorithm would be to use a fast and ro- 
bust optical flow algorithm, such as the one presented 
in [8], to select the regions where the optical flow is 
small and then constrain the search for features to these 
regions. 

Although the resulting stabilized sequence is ap- 
propriate for visualization purposes, it cannot be di- 
rectly applied to segment IMOs from simple difference 
of frames, mainly due to the clutter produced by the 
regions that do not fit the 2D motion assumption and 
small misalignments of the registration process, but ob- 
serve that the moving object present in the sequence is 
clearly segmented. Qualitative methods [10] and tem- 
poral filters combined with velocity tuned filters [9] 
were suggested to detect IMOs in such conditions. 

4. Conclusion 

We presented in this paper a fast electronic digital 
image stabilization system based on a two-dimensional 
feature-based multi-resolution motion estimation algo- 
rithm, that tracks a small set of features to estimate 
the motion of the camera. Stabilization is achieved 
by combining all motion from a reference frame and 
subtracting this motion from the current frame. The 
system was implemented in a Datacube Max Video 200 
board connected to a SUN SPARCstation 20/612 via 
a VME bus adaptor. Preliminary tests using video se- 
quences recorded from a vehicle moving on rough ter- 
rain demonstrate the robustness of the system, that is 
able to process 15 frames per second and handle dis- 
placements of up to ±21 pixels between consecutive 
frames. Although these figures might be compatible or 
better than current stabilization systems, it is some- 
times not enough to stabilize sequences taken in off- 
road conditions. 

Further research effort is under way to integrate sta- 
bilization with other image processing applications like 
video compression and automatic detection, tracking, 
and recognition of IMOs. 
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